Papers with parameter sharing
Joint models for NLP (D18-3)
Copied to clipboard
| Challenge: | This tutorial reviews main approaches to joint modeling for statistical and neural methods. |
| Approach: | This tutorial reviews main approaches to joint modeling for both statistical and neural methods. |
| Outcome: | This tutorial reviews main approaches to joint modeling for statistical and neural methods. |
YANMTT: Yet Another Neural Machine Translation Toolkit (2023.acl-demo)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) is an end-to-end approach that provides stateof-the-art results for a variety of language pairs. |
| Approach: | They propose to build an open-source neural machine translation toolkit on top of HuggingFace's Transformers library and use it for pre-training and fine-tuning sequence-to-sequence models. |
| Outcome: | The proposed toolkit is built on top of the HuggingFace Transformers library and provides advanced features such as document/multi-source NMT, simultaneous NMT and mixtures-of-experts. |
From Research to Production and Back: Ludicrously Fast Neural Machine Translation (D19-56)
Copied to clipboard
Young Jin Kim, Marcin Junczys-Dowmunt, Hany Hassan, Alham Fikri Aji, Kenneth Heafield, Roman Grundkiewicz, Nikolay Bogoychev
| Challenge: | Using the dominating submissions to the previous edition of the shared task, we develop improved teacher-student training via multi-agent dual-learning and noisy backward-forward translation for Transformer-based student models. |
| Approach: | They propose to use multi-agent dual-learning and noisy backward-forward translation to improve teacher-student training for Transformer-based student models. |
| Outcome: | The proposed model outperforms submissions to the previous edition of the WNGT efficiency shared task by 4 BLEU points and 10 BLUE points respectively. |
Deploying Multi-task Online Server with Large Language Model (2025.coling-industry)
Copied to clipboard
| Challenge: | In the industry, numerous natural language processing tasks are deployed online . traditional approaches tackle each task separately by its own network and pipeline . |
| Approach: | They propose a three-stage multi-task learning framework for large language models . it involves task filtering, fine-tuning on high-resource tasks, and finally fine- tuning on all tasks . |
| Outcome: | The proposed framework reduces up to 90% of overhead while reducing latency and resource usage. |
A Multi-lingual Multi-task Architecture for Low-resource Sequence Labeling (P18-1)
Copied to clipboard
| Challenge: | Existing studies have shown that multi-task learning can boost the performance of related tasks such as MT and abstractive text summarization. |
| Approach: | They propose a multi-lingual multi-task architecture to develop supervised models with a minimal amount of labeled data for sequence labeling. |
| Outcome: | The proposed architecture achieves 4.3%-50.5% absolute gains compared to mono-lingual model . the proposed model is particularly effective in low-resource settings . |
Efficient Hierarchical Domain Adaptation for Pretrained Language Models (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to allow domain adaptation to diverse domains are expensive and require continuing training in-domain. |
| Approach: | They propose a method to permit domain adaptation to many diverse domains using a computationally efficient adapter approach. |
| Outcome: | The proposed method allows domain adaptation to many diverse domains while avoiding negative interference between unrelated domains. |
A Compact and Language-Sensitive Multilingual Translation Method (P19-1)
Copied to clipboard
| Challenge: | Existing paradigms for multilingual neural machine translation do not make full use of language commonality and parameter sharing. |
| Approach: | They propose a multilingual neural machine translation paradigm with one encoder-decoder model that makes full use of language commonality and parameter sharing. |
| Outcome: | The proposed method outperforms strong standard multilingual translation systems on WMT and IWSLT datasets. |
Task-Aware Specialization for Efficient and Robust Dense Retrieval for Open-Domain Question Answering (2023.acl-short)
Copied to clipboard
| Challenge: | Existing dense retrieval models are parameter-inefficient and underperform sparse counterparts. |
| Approach: | They propose a task-aware specialization for dEnse Retrieval architecture that enables parameter sharing by interleaving shared and specialized blocks in a single encoder. |
| Outcome: | The proposed architecture surpasses BM25 on questions and passages using 60% of the parameters as bi-encoder dense retrievers. |
Task-wrapped Continual Learning in Task-Oriented Dialogue Systems (2025.findings-naacl)
Copied to clipboard
| Challenge: | Continual learning is vital for task-oriented dialogue systems (ToDs), but its performance is limited by training separate adapters for each task, preventing global knowledge sharing. |
| Approach: | They propose a framework that employs task-wrapped Adapters to learn global and task-specific information through parameter sharing. |
| Outcome: | The proposed framework outperforms AdapterCL in 37 domains while using only 46% of the parameters. |
Transfer and Multi-Task Learning for Noun–Noun Compound Interpretation (D18-1)
Copied to clipboard
| Challenge: | In computational linguistics, nounnoun compound interpretation is approached as an automatic classification problem. |
| Approach: | They empirically evaluate the utility of transfer and multi-task learning on a challenging semantic classification task. |
| Outcome: | The proposed methods improve the accuracy of a neural classifier and its F1 scores on the less frequent, but more difficult relations. |
Adaptive Parameterization for Neural Dialogue Generation (D19-1)
Copied to clipboard
| Challenge: | Existing models of open-domain dialogue generate responses based on sequence-to-sequence paradigms. |
| Approach: | They propose an Adaptive Neural Dialogue generation model which manages various conversations with conversation-specific parameterization. |
| Outcome: | The proposed model performs better on a large-scale conversational dataset. |
Multimodal Phased Transformer for Sentiment Analysis (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to model multimodal sentiment analysis are limited due to their complexity and memory footprint. |
| Approach: | They propose a multimodal Sparse Phased Transformer to reduce self-attention complexity and memory footprint. |
| Outcome: | The proposed method achieves comparable or superior performance with a 90% reduction in the number of parameters. |
Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing approaches to retrieve hard negative sentences are limited in the scale of the dataset thus fail to identify negative samples of high difficulty for every image. |
| Approach: | They propose to use a model to generate synthetic negative sentences with higher difficulty by masking and refilling the images and performing word discrimination and word correction tasks to improve retrieval and generation. |
| Outcome: | The proposed model generates synthetic negative sentences with higher difficulty on MS-COCO and Flickr30K and is robust and faithful to state-of-the-art training. |
TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs (2026.findings-eacl)
Copied to clipboard
Minjae Lee, Wonjun Kang, Byeongkeun Ahn, Christian Classen, Kevin Galim, Seunghyuk Oh, Minghao Yan, Hyung Il Koo, Kangwook Lee
| Challenge: | Large Vision Language Models (LVLMs) are advanced models that process multiple modalities, such as images, audio, and video, alongside text. |
| Approach: | They propose to use a method to generate and verify draft tokens in parallel . they compare existing methods with small draft models and observe performance fluctuations . |
| Outcome: | The proposed method achieves an average walltime speedup of 1.74 over autoregressive decoding and a 5% improvement over single drafting methods. |
NeuSTIP: A Neuro-Symbolic Model for Link and Time Prediction in Temporal Knowledge Graphs (2023.emnlp-main)
Copied to clipboard
| Challenge: | Temporal Knowledge Graphs (KGs) are factual information repositories where a fact is associated with a time interval. |
| Approach: | They propose a temporal NS model for knowledge graph completion that performs link prediction and time interval prediction in a TKG. |
| Outcome: | The proposed model shows competitive performance on link prediction and time prediction. |
Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each. |
| Approach: | They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks. |
| Outcome: | The proposed setup improves on the state-of-the-art for a total of 15 languages. |
WARP: Word-level Adversarial ReProgramming (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to transfer learning from pretrained language models are frozen and a task-specific head is trained on top of them. |
| Approach: | They propose an alternative approach that trains one or more task-specific layers on top of the language model. |
| Outcome: | The proposed approach outperforms existing methods on the GLUE leaderboard with just 32 training samples. |
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for quantizing weights and activations of large language models suffer from non-negligible accuracy drops, especially on massive multitask language understanding. |
| Approach: | They propose a weight-activation quantization method that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices. |
| Outcome: | The proposed method reduces the complexity of the weight-activation quantization techniques while achieving high throughput and reducing inference costs. |
HYDRA: A Multi-Head Encoder-only Architecture for Hierarchical Text Classification (2025.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art approaches rely on complex components like graph encoders, label semantics, and autoregressive decoders. |
| Approach: | They propose a multi-head encoder-only architecture for hierarchical text classification that treats each level as a separate classification task with its own label space. |
| Outcome: | The proposed architecture matches or exceeds state-of-the-art methods on four benchmarks. |
LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing (2023.findings-acl)
Copied to clipboard
| Challenge: | Deploying Transformer networks on resource-constrained edge devices is challenging. |
| Approach: | They propose a low-rank factorization initialized by SVD-based weight transfer and parameter sharing to compress and accelerate Transformer networks. |
| Outcome: | The proposed method achieves similar performance to the baseline Transformer with 3.8 times and 1.8 times fewer parameters and achieves 2.3 times speedup and 1.5 times speed up respectively. |
Efficient Transformer Parameter Reuse via Zero-Token Mechanism (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to scaling up parameter counts are impractical for users with limited computational resources. |
| Approach: | They propose a decoupled parameter cycling strategy that employs a head-tail decoupling strategy to decouple the first (head) and last (tail) layers from the parameter cycling process. |
| Outcome: | The proposed approach achieves superior performance under strict parameter constraints and significantly reduces computational overhead via early exits. |
Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments. |
| Approach: | They propose a method to enhance the inference efficiency of parameter-shared PLMs by pre-training models that can achieve even greater acceleration. |
| Outcome: | The proposed method improves inference efficiency on autoregressive and autoencoding models. |
Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that pixel representations can be finetuned across scripts without vocabulary extensions, adapters, or transliteration. |
| Approach: | They propose to use pixel representations to train multilingual machine translation models . they explore parameter sharing within and across scripts to better understand where they lead to positive transfer . |
| Outcome: | The proposed model improves on two multilingual datasets with different language coverage compared to subword embeddings . the proposed model can be finetuned cross-lingually or to unseen scripts, and is more data-efficient than other alternatives such as vocabulary expansion . |
SharVeT: Similarity-aware Parameter Sharing with Vector-based Tuning for Efficient LLM Compression (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for parameter sharing rely on naive grouping and fail to correct sharing-induced discrepancies. |
| Approach: | They propose a parameter sharing framework that performs similarity-based grouping to ensure accurate sharing and allocates parameters adaptively to preserve diversity within each group. |
| Outcome: | The proposed framework outperforms existing methods, achieving 32.1% lower perplexity and 23.3% higher few-shot reasoning accuracy. |